Skip to content

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525

Open
indietyp wants to merge 4 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres
Open

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525
indietyp wants to merge 4 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres

Conversation

@indietyp
Copy link
Copy Markdown
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Prepares the MIR execution pipeline for consumption by the postgres compiler. The execution analysis now produces a complete IslandGraph (not just a flat island list), the placement solver and island placement accept external allocators, and the traversal system gains the TraversalPathBitMap and as_symbol() APIs that the SQL generator needs. Also adds a backend switch cost to the terminator placement so cross-backend transitions are no longer free.

🔍 What does this change?

Execution analysis (pass/execution/mod.rs):

  • run() becomes run_in() with an explicit allocator parameter, returning ExecutionAnalysisResidual (assignment + island graph). The island graph is now constructed as part of the analysis rather than left to the caller.
  • Adds run_all_in() which runs the execution analysis over all graph-read bodies in a DefIdSlice.

Traversal system (traversal/mod.rs, traversal/entity.rs):

  • TraversalPathBitMap: per-vertex-type collection of TraversalPathBitSets with pointwise lattice operations. The postgres compiler uses this to track which paths each island accesses across all vertex types.
  • TraversalPath::as_symbol(): returns a static symbol for each path variant, used as SQL column aliases so the interpreter can locate result columns by name.
  • TraversalPathBitSet::vertex(): returns the vertex type for a bitset.
  • EntityPath::as_symbol() and EntityPath::column_name(): per-path SQL identifiers.

Terminator placement (terminator_placement/mod.rs):

  • TransMatrix gains AddAssign for element-wise saturating addition.
  • Adds backend_switch_cost() which encodes a fixed overhead for cross-backend transitions (Postgres to Interpreter: 8, Interpreter to Embedding: 4, etc.). Previously cross-backend transitions had zero inherent cost, so empty blocks were arbitrarily assigned to the interpreter even when staying on postgres was free.

Placement solver (placement/solve/):

  • PlacementSolver::run() becomes run_in() with allocator parameter.
  • CSP solver internal allocations use the provided allocator.

Island graph (island/graph/mod.rs):

  • IslandGraph::new_in() now takes an allocator for its output storage.

Pretty printer (pretty/text.rs):

  • TextFormatAnnotations gains annotate_basic_block() and BasicBlockAnnotation associated type, plus a blanket impl for &mut T.

Builder (builder/rvalue.rs):

  • Adds RValueBuilder::opaque_entity_uuid() convenience constructor for the common pattern of extracting an entity's UUID.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The backend_switch_cost() values (8, 4, etc.) are hand-tuned heuristics. They correctly prevent the solver from arbitrarily switching backends for empty blocks, but a proper cost model would derive these from measured overhead.

🛡 What tests cover this?

  • Updated execution pass end-to-end tests (execution/tests.rs) including entity_uuid_equality, mixed_postgres_embedding_interpreter, projection_and_apply_splits
  • Updated statement placement tests for interpret and postgres backends (eq_opaque_entity_uuid)
  • Updated terminator placement and island tests
  • Placement solver tests updated for allocator API

❓ How to test this?

cargo nextest run -p hashql-mir

feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@cursor
Copy link
Copy Markdown

cursor Bot commented Mar 8, 2026

PR Summary

Medium Risk
Medium risk because it changes core execution-placement APIs and the cost model (backend switch overhead), which can shift target assignments and island scheduling across the pipeline.

Overview
Refactors MIR execution analysis to be allocator-parameterized and to return a richer ExecutionAnalysisResidual containing the per-block target assignment plus a fully built IslandGraph, and adds run_all_in to analyze multiple graph-read bodies in one pass.

Extends island/traversal infrastructure for SQL generation: IslandGraph gains helpers (members, contains, find), traversal paths gain stable as_symbol() identifiers, and a new per-vertex TraversalPathBitMap lattice is introduced.

Updates placement/terminator cost modeling and plumbing: PlacementSolver::run_in now allocates via a caller-provided allocator, terminator placement adds a fixed cross-backend backend_switch_cost() (applied via TransMatrix +=), and tests/benchmarks/snapshots are refreshed (including new opaque-type equality coverage using an opaque rvalue builder/macro support and pretty-printer basic-block annotations).

Reviewed by Cursor Bugbot for commit 44ba0c2. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment May 11, 2026 11:07am
petrinaut Ready Ready Preview May 11, 2026 11:07am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview May 11, 2026 11:07am
hashdotdesign-tokens Ignored Ignored Preview May 11, 2026 11:07am

Copy link
Copy Markdown
Member Author

indietyp commented Mar 8, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: Prepares the HashQL MIR execution pipeline for Postgres compilation by returning a full island dependency graph and exposing traversal/placement APIs needed by the SQL generator.
Key Changes:

  • Execution analysis now takes an explicit allocator (run_in) and returns ExecutionAnalysisResidual (assignment + IslandGraph); adds run_all_in for batch analysis.
  • Traversal adds TraversalPathBitMap, TraversalPath::as_symbol(), and EntityPath::as_symbol()/column_name() for stable SQL column aliases.
  • Placement solver and island placement APIs accept external allocators and route CSP internal allocations through the provided bump allocator.
  • Terminator placement introduces a fixed backend_switch_cost() so cross-backend transitions have non-zero overhead (avoids “free” backend flips).
  • Island graph gains helper queries (members, contains, find) and is now built/resolved within execution analysis.
  • Pretty printer supports per-basic-block annotations; MIR builder gains an opaque aggregate constructor and new tests cover opaque UUID equality.
Tests: Updates/extends execution, statement placement, terminator placement, and allocator-related solver snapshots (incl. entity_uuid_equality). Notes: Backend switch costs are heuristics intended to be refined by a measured cost model.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 64.24581% with 128 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.31%. Comparing base (959fba1) to head (44ba0c2).

Files with missing lines Patch % Lines
...cal/hashql/mir/src/pass/execution/traversal/mod.rs 0.00% 44 Missing ⚠️
libs/@local/hashql/mir/src/pretty/text.rs 21.62% 28 Missing and 1 partial ⚠️
.../hashql/mir/src/pass/execution/traversal/entity.rs 0.00% 28 Missing ⚠️
libs/@local/hashql/mir/src/pass/execution/mod.rs 44.00% 14 Missing ⚠️
.../hashql/mir/src/pass/execution/island/graph/mod.rs 35.29% 11 Missing ⚠️
.../mir/src/pass/execution/placement/solve/csp/mod.rs 90.00% 0 Missing and 1 partial ⚠️
libs/@local/hashql/mir/src/pass/execution/tests.rs 99.07% 1 Missing ⚠️
Additional details and impacted files
@@                                          Coverage Diff                                          @@
##           bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query    #8525    +/-   ##
=====================================================================================================
  Coverage                                                                62.31%   62.31%            
=====================================================================================================
  Files                                                                     1354     1354            
  Lines                                                                   137003   137313   +310     
  Branches                                                                  5792     5793     +1     
=====================================================================================================
+ Hits                                                                     85372    85567   +195     
- Misses                                                                   50725    50839   +114     
- Partials                                                                   906      907     +1     
Flag Coverage Δ
rust.hashql-compiletest 29.63% <ø> (ø)
rust.hashql-mir 92.08% <64.24%> (-0.37%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 56 skipped benchmarks1


Comparing bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres (44ba0c2) with bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (959fba1)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 6ef6d13 to ef9b858 Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5d95ba7 to 1aa0f1c Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 1aa0f1c to 87024df Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 0bd3b25 to a675701 Compare April 29, 2026 13:08
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from bc50eb0 to 8b6ee87 Compare April 29, 2026 13:08
Comment thread libs/@local/hashql/mir/src/pass/execution/tests.rs
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5552cbb. Configure here.

Comment thread libs/@local/hashql/mir/src/pass/execution/mod.rs
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5552cbb to 60da845 Compare May 4, 2026 11:34
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 60da845 to fd9a3a8 Compare May 8, 2026 08:33
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 4f33a35 to 959fba1 Compare May 11, 2026 10:58
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from fd9a3a8 to 44ba0c2 Compare May 11, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants